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Abstract — We describe the conclusions of a technology and 
communities survey supported by concurrent and follow-on 
proof-of-concept prototyping to evaluate feasibility of defining a 
durable, versatile, reliable, visible software interface to support 
strategic modularization of test software development. The 
objective is that test sets and support software with diverse 
origins, ages, and abilities can be reliably integrated into test 
configurations that assemble and tear down and reassemble with 
scalable complexity in order to conduct both parametric tests and 
monitored trial runs. The resulting approach is based on 
integration of three recognized technologies that are currently 
gaining acceptance within the test industry and when combined 
provide a simple, open and scalable test orchestration 
architecture that addresses the objectives of the Automation 
Hooks task. The technologies are automated discovery using 
multicast DNS Zero Configuration Networking (zeroconf), 
commanding and data retrieval using resource-oriented Restful 
Web Services, and XML data transfer formats based on 
Automatic Test Markup Language (ATML). This open-source 
standards-based approach provides direct integration with 
existing commercial off-the-shelf (COTS) analysis software tools. 

Index Terms — Software standards. Test equipment. Test 
facilities. Testing. Software management. Software reusability 

I. Introduction 

N ASA’s Constellation Program identified an opportunity 
to reduce out-year operating costs for system and sub- 
system integration test operations through automation-assisted 
test choreography and data orchestration. There arc 
complimentary opportunities to improve scientific research 
and engineering development workflows. 

Essentially, the opportunity is that even for run-once and 
investigative testing, COTS and even custom hardware is 
configured and monitored by users through keyboard-and- 
mousc software packages. If data from these 
i fe par a tehctcrogcncous modules could be harvested through a 
robust, open standard based infrastructure, the data products 
could be formed more quickly, accurately, and thoroughly. 
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and results correlated more powerfully — by comparison with, 
say, having users type data from screens into spreadsheets or 
collect and transfer data files in an ad-hoc fashion. 

The application scenarios anticipated are not high volume 
or highly repetitive. Automating development of the test 
procedures themselves from requirements is not a significant 
area of interest. Much of the potential for saving is related to 
discovery of module data requirements and aggregation of test 
data in integrated scenarios containing a changing assortment 
of highly complex and coupled modules. 

There arc many technical challenges to address, but first 
one must confront the organizational challenge in that the 
software modules available at a system integration test, for 
example, are of diverse origins and developed on 
heterogeneous platforms. 

Having accomplished the aggregation of coincident 
observations, one can further imagine storing and restoring the 
configuration of the test bed using read/write interfaces, and 
ultimately it could be possible to repeat test sequences and 
overlay data. 

Quiescent, continuous, and event-driven test cycles are 
anticipated. Scripted flows are presumed built on less 
structured fault-isolation or experimentation test flows. 

The concept of operations imposes some constraints 
required to enable data correlation. These include time 
synchronization mechanisms and resources, data indexing, 
labeling data with metadata, and encouraging the use of 
widely understood self-describing data formats. 

II. Criteria and Field of Choices 

A concept of operations was proposed, and then distilled 
down to a set of "guiding principles” which could be used for 
evaluating different approaches. These principles included: 

Non-proprietary, with multiple vendors. A proprietary or 
single-vendor interface could not achieve universal 
penetration into varied developments and could present a 
single-point-of-failure risk to the Program. 

Widespread usage, with active user communities. Our 
intention was not to reinvent the interface and associated 
toolsets, but rather to find and adopt (adapt) already widely 
supported technologies. 

Supported in the Test industry. Interfaces with existing 
support in NASA. DoD, and consumer communities and test 
COTS products were given affirmative weight. 

Multiple sources of ready development tools. Software 
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interfaces supported by a family of open source tools provide 
rapid deployment. 

Language and OS independent. Interfaces that are tied to 
specific operating systems or development environments only 
solve part of the problem, and are vulnerable to accelerated 
obsolescence. 

Having described what we were seeking, we surveyed test 
communities at NASA. DoD, and in industry, and also 
considered plug-and-play consumer interfaces. We considered 
that our software elements could use simulation interfaces, or 
instrumentation interfaces, or web services interfaces. 
Fundamentally the difference between requirements for a test 
software interface and a simulation software interface is that 
the modules do not need to interact. 

HI. Summary of Study Observations 

A. Existing End-to-End Infrastructures 
Several existing end-to-end simulation and test infrastructures 
were investigated in an attempt to find an out-of-thc box 
capability that could be used to meet the trade study criteria. 

High Level Architecture (HLA) has been used in the 
Constellation program as an architecture for distributed 
dynamics simulations. It was examined briefly but it was 
quickly decided that the overhead associated with its run time 
infrastructure and simulation federate organization made it 
unattractive as a test orchestration infrastructure. 

The Test and Training Enabling Architecture (TENA) is a 
DoD initiative aimed at distributed simulation and test 
applications. It is geared to supporting test ranges and 
facilities. TENA seemed to require middleware that appeared 
to be single-source. In addition, it is based on CORBA, an 
object-based messaging protocol that has been declining in 
popularity because of its complexity and historical difficulty 
penetrating firewalls. Interest in the wider community has 
shifted from CORBA and its competitor DCOM to Web 
Services which are discussed later. 

13. Established Test Software Interfaces 

A promising early candidate was the LX1 interface, and 
ultimately we adopted several features of this interface. The 
interface was discarded because tools for development of LX1 
hosts were not readily available. 

Investigation of the DoD Automatic Test Systems (ATS) 
Open Systems approach lead to interest in the Automatic Test 
Markup Language as a data format. Many of the approaches of 
the ATS Open Systems approach were compatible with our 
"guiding principles”. Virtual Instrument Software Architecture 
(VISA) and Interchangeable Virtual Instruments (1VI) 
technologies were determined to be too low-level for our goal 
and available drivers appeared to be limited to the Windows 
OS. In addition, these technologies did not appear to be widely 
used outside the Automatic Test Equipment industry. 

NASA’s Constellation program was also developing an 
interface for avionics test orchestration. Software and 
Avionics Test Orchestration Command and Messaging 
(SATOCM). We did exchange observations with this group, 
and although there remain differences in emphasis both teams 


believed it would be possible to achieve convergence. This 
interface was designed to simplify test script-writing using 
Python. Our study was not able to adopt it because its current 
incarnation contains a complex command set delivered by a 
custom binary messaging mechanism. . tvas rejected as - isby our 
study be cause it v i el a t es- m any of our guiding p r i n c ipl es . 

C. Discover)’ Protocols 

Universal Plug and Play (UPnP) was evaluated against 
Zeroconf. Both were strong candidates, but we perceive that 
UPnP is fading, and Zeroconf has existing heritage in the test 
community through its use in LX1. 

D. Messaging Protocols 

Several message oriented protocols and middleware APIs at 
several different levels of complexity were considered. Some, 
like Java Message Service (JMS) were not language neutral. 
Some like Advanced Message Queuing Protocol (AMQP) 
introduced complexity by solving problems we did not have. 
The Simple Object Access Protocol (SOAP) web sendees 
protocol was chosen for initial prototyping because it satisfied 
our evaluation criteria and fit well with ATML which was also 
of interest. There is a wddc variety of tools and 
implementations available including many open-source 
packages. It is also widely used and accepted in many 
industries. 

A functional prototype was implemented using SOAP. Many 
parts of the SOAP implementation, however, were found to be 
complex in the face of limited prototyping resources. For 
example, Web Services Description Language (WSDL) files 
were found to be complex to create and maintain. Different 
implementations of SOAP were found to be incompatible 
without detailed attention to configurations and options. No 
insurmountable problems were encountered but eventually a 
decision was made to prototype an alternative resource 
oriented or Representational State Transfer (RESTtul) style of 
web services. The level of simplification, elegance and 
increased case of implementation was so striking that 
ultimately when faced with building a prototype using limited 
resources we opted for the RESTful approach. This choice 
affected not just the messaging protocol but the overall 
architecture and division of responsibilities between test 
orchestration software and individual test set interfaces. 

Many of RESTful features such as Uniform Interface, 
stateless server restrictions, and cachcable responses 
contributed to robustness and enhanced visibility of the test 
protocol. Also, the perspective on commanding test 
equipment changed from remote-procedurc-call (RPC) based 
to resource based which was found to result in gains in 
elegance and simplicity. 

E. Command Sets 

Test execution interfaces have a long history of using verb- 
based command sets, including HP BASIC, ATLAS, and 
SCP1. NASA’s SATOCM command set was intended to 
simplify script-writing, and initially we planned to implement 
a subset of SATOCM commands. 

The RESTful style architecture primarily uses a small subset 
of standard HTTP commands such as GET, PU T, POST, and 


1569299649 


3 


DELETE directly. The richness of the interface is then 
captured as resources that arc manipulated using these 
standard HTTP commands. This approach replaces the 
requirement to create a traditional RPC'-based set of 
commands with the requirement to design appropriate 
resources to represent required test concepts. In prototyping, 
the resource-based approach was found to result in a simpler 
and more transparent infrastructure. 

The command and error message sets already provided by 
HTTP are compact, powerful, and understood by a large 
collection of off-the-shelf software. The command set is 
compact and powerful, and the error message set is rich. 
Security and data compression solutions arc innate. 

F. Delia Interface Protocols 

We evaluated the architecture of having software modules 
write directly to a designated database interface without an 
intermediary. Scalability and robustness were identified 
obstacles. To make a successful interface, a completed 
software module must be able to create its own tables and 
write data to them without further changes to the platform to 
accommodate different database vendors or other changes in 
database technology. JDBC was entertained as meeting this 
objective, but limits the usability of the module by requiring 
each software module to interface with Java. An ODBC 
driver approach was evaluated but required specialized 
software to be installed and maintained on each client. An 
ODBC'bridgc driver approach eliminates the client software 
issue, but introduces an issue with proprietary software and a 
sole-source provider. It was determined that this type of SQL- 
oriented middleware merely transfers the maintenance 
problem to another vendor who must then be required. 

The solution that worked best in prototyping and met the 
goals of the study was to use the resource oriented interface to 
serve data. An unexpected side benefit was that data resources 
could be accessed by web-ready off-the-shelf software. For 
example, prototypes built on modular open source software 
have demonstrated that this interface is already natively 
accessible to web browsers and to Excel. 

Data log requests are submitted by the orchcstrator and each 
testset module makes locally buffered data accessible through 
a resource interface for that data log request. This approach 
allows data to be logged with arbitrary resolution . Tight 
coupling mechanisms such as data pushes, reflective memory, 
or messaging middleware are unnecessary because - and 
alignment occurs after the observations arc aggregated. 

G. Data Formats 

Software written by hardware engineers often will write data 
using comma separated value (CSV) or tab separated value 
(TSV) formats. These formats arc easy to generate, widely 
supported by tools, and are decades old. There are, however, 
many format variations including how commas are handled 
within a data file. This can be particularly troublesome for 
countries that use commas as the decimal separator, in 
addition there are no rccommcndable approaches for 
incorporating meta-data. There are also some operating 
system differences. 


Binary formats are very system-dependent, although they 
can be supplemented with descriptive XML metadata files. 

The SQL statement format was also considered, but the 
availability of XML-enablcd databases diminishes the appeal 
of this option. It was further identified that different database 
vendors have different interpretations of the SQL standard. 

XML allows sufficient metadata to be included so that 
database tables can be automatically created, standardizes the 
dat adatc -time format, and allows further information like 
theory of operation (help-text) for a parameter to be captured. 
ATML provides an XML language that standardizes 
information exchange for many kinds of data and mcta-dala 
we are interested in capturing, and is also becoming 
represented in test industry products. An alternative schema, 
NASA Exploration Information Ontology Model (NExIOM), 
was discarded only because it has a limited following. The 
authors hope that NASA can participate in the further 
development of ATML. 

IV. Conclusion 

Combining RESTful principles with Zcroconf and ATML 
formed a powerful, versatile, rugged interface that met all of 
the study criteria. The combination was found to provide a 
simple, elegant, and easy to use infrastructure for test 
orchestration. Prototypes have already demonstrated 
connectivity with LabVIEW, NASA's Trick Simulation 
Development Environment, and the Engineering DOUG 
Graphics for Exploration (EDGE) software used for 3D 
graphics rendering in Constellation training and test facilities. 
Prototypes have been hosted in various distributions of the 
Linux operating system and in Windows XP and Vista. 
Distributed and co-hosted topologies have been demonstrated, 
and multiple copies of modules arc distinguishable. The 
interface has been demonstrated with both simulations and 
hardware and has been used to orchestrate a distributed Orion 
abort-to-orbit test scenario using JSCs Avionics Integration 
Enivironment (A1E) facility and the Reconfigurable Cockpit 
Simulation Facility. It is being integrated with test hardware 
in the Kedallion avionics facility and the Electronics Systems 
Test Lab (ESTL) at Johnson Space Center. Wc currently rate 
this interface as Technology Readiness Level 4. 
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Don't Rush to Directly Solve the Problem, 
Create a Machine to Solve the Problem 
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NIH began the Human Genome Project in 1993 
and expected to finish in 2003 (10 years) at a cost 

of $3ooB.ji fii!iii.if| B f|jj|Wii. 

Dr. Venter left NIH in 1998, founded Celera f*Ma-BSf 
Genomics, and completed sequencing of the 
human genome in 2000 (2 years) at a cost of $3B. 
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Not just 

stripcharts 
Analyze before 
teardown 


Real-time steering 


IEEE Standard for Automatic Teat 
Markup language (ATMl) for 
Exchanging AutomaDc Tost Eautpment 
and Teat Inlormatlon via XML 


Archive product 


Post-analysis 


Ijlt • : 


i II — -i-x 



( 


Engineering 

Report 




Requirements 

Basic Functional Requirements 


• Discovery 

- TFDM chooses LTE's required for test configuration 

- LTE resources/command set available to TFDM upon discovery 

• Initialize, configure or return LTE's to known configurations 

• Provide LTE status to TFDM directly from LTE or through a database 

• Middleware to buffer LTE's from changes in database vendors/versions 

• Support parametric as well as time-based test objectives 

• Don't preclude 

- multiple tests on the same network 

- other test equipment on same network 

- ability of LTE's to join and leave test config as required by test conductor 

- Multiple databases 
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Technology Survey and Trade Study 


Non-proprietary with 
multiple vendors 
Widespread, active 
user communities 
Supported in the Test 
industry 

Multiple sources of 
ready development 

tools 

Language and OS 
independent 



Lung-te, m AvdHability, No Obligation to Buy 
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industry Bust Practice 
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• Surveyed NASA, Test COTS, DoD, and 
Consumer communities for viable approaches 

• Down-selected based on "guiding principles" 












Test Technology Survey 

Test Technology Survey 


• Performed a survey (web, interviews, tour) to identify 
methods/protocols being used for discovery and test control 
in relevant test communities. 

• Looked at two NASA examples 

- DSIL 

- ISIL 

• Looked at two DoD examples 

- TENA 

- DoD ATS 

• Looked at Commercial Products 

- Nl Labview and Test Stand Test Software 

- IXIA IP Test Automation 
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What did the AHA study conclude? 


• Discovery: advertised resources using 
Zeroconf (DNS-SD, mDNS, link-local) 

• Data Representation: XML and some ATML 

• Control/Status: REST architecture over HTTP 



Discovery 


• Assembling the automation can't take extra 
time, on top of assembling the testbed, so 
simple plug-and-play discovery is highly 
desirable. 

• Zeroconf has heritage in LXI, LabVIEW, and 

consumer products. ^ ■ vJ 

Modules don't need to be responsible for 
tunneling discovery through firewalls, this can 
be handled using proxy method or unicast. 


Prototyping/Labview Interface 


Dynamic Port Assignments 



*29 Signal Source. lvlib:Slgr>al Source. vi 


Fit Edl Cferate Tods Mom Hdp 


hllp ://J5C-20080B560 .jsc. nasa. [pv/Sgnd%20fburcc .Mb_5gnd%205ajnc.Nml 


Status_teta 


Filet 


C:rfiguro 


Run I Rrmole 


[nstiuctiore 
1. Pun this saver Vi 

Z. Run any of the (lent VTs r diwts. Mb and 
the/ shtxJd pop date in the cierC \& 


Freguenr/ Ccrtrd 


Hire Stamp 


dect Wine 


Side Ccrb-d 


|5IG_GEW 


Stiinj Ccrtnd 


Side Contiol 


Enurr 


Omdltuite 


Tine 


O.OE+G 20E-2 


Plug-and-Play 
discovery enables 
dynamic IP address and 
Port assignments. 


Fie Edit Operate Tools Wjndo/i (Trip 










Conliqj'o Run Reimls About 








'-~s> 


If a computer 
goes down, give it 
to a tech and use 
a different one! 


a i. 


I la SigidSouiceM,. 


G 5iA'¥ , OH0 l ® 5;05CM 


Nov 12, 2009 


Test Automation Hooks Study 
EV- METECS 


11 





The XML File Format 


The Good 

• Provides a method for 
thorough description: good 
naming, units, representation 

• Record presentation format is 
defined separately (XSL style 
sheet) 

• Provides many inherently 
standard data representations, 
including timestamps 

• Provides a method for 
representing hierarchical data 

• The file format is standardized 

• Adoption is spreading 


The Bad 

• Data cannot be appended to 
an XML file, the entire file 
should be parsed and 
rewritten and the data must 

be inserted. 

. "Free" tools now are crude 

- MS XML Notepad, Excel 

• Good tools now are not free; 
JSC/EA doesn't have them 

- XML Fox $280/seat, 
$1200/corp 

- Data Direct's Stylus Studio 
~$25k 

• It's not as "human readable" 
as you may have heard 


Opening an XML file in Excel 


I? JitterController-Group3.xml - Notepad 


File Ed* Format View Help 


</root> 

< 


</row> 


<source_3itctr1. 
<Source_JitCtr1. 
<source_Jitctr1. 
<source_Jitctr1. 
<source_Ji tetri. 
<source_Ji tetri. 
<source_Ji tetri. 
<source_Jitctr1. 
<sour ce_J i tet r 1 . 
<Source_Ji tetri. 
<Source_JitCtr1. 
<Source_3i tetri. 
<Source_3itetr1. 
<Source_Ji tetri. 


usER_Testl_crestfactor_FJ_ratio_FLOAT>NaN</source_ditctr1_usER_Testl_crestfactor_FJ_ratio_FLOAT> 

.USER_Testl_ineasPhj_rms_pct_FLOAT>NaN%</source_1itCtr1_uSER_Testl_nieasPha_rms_pct_FLOAT> 

.usER_Testl_meas Ph3_pk_pct_FLOAT>NaNX</sour ce_3 i tet r 1 _usER_Testl_meas Ph J_pk_pct_FLOAT> 

,usER_Testl_crestf actor_ph3_ratio_FLOAT>MaN</source_litctr1_usER_Testl_crestfactor_Ph3_ratio_FLOAT> 

.usER_Testl_iitterxfer_ratio_FLOAT>NaN</source_3itctr1_usER_Testl_jitterxfer_ratio_FLOAT> 

,usER_Testl_BERdegr_dB_FLOAT>NaN</source_3itctr1_usER_Testl_BERdegr_dB_FLOAT> 

.usER_Test2_measF3_rms_pct_FLOAT>NaN%</source_3itctr1_usER_Test2_measF3_rms_pct_FLOAT> 

.LlSER_Test2_measF3_pk_pct_FLOAT>NaM%</Source_3itCtr1_usER_Test2_measF3_pk_pct_FLOAT> 

.U5ER_Test2_crestfactor_F3_ratio_FLOAT>MaN</source_3itCtr1_usER_Test2_crestfactor_F3_ratio_FLOAT> 

.LlSER_Test2_measPh3_rms_pct_FLOAT>l. 648</Source_3itCtr1_uSER_Test2_tneasPh3_rms_pct_FLOAT> 

.U5ER_Test2_measPh3_pk_pct_FLOAT>NaN%</source_3itCtr1_uSER_Test2_ineasPh3_pk_pct_FLOAT> 

.USER_Test2_crestfactor_Ph3_ratio_FLOAT>NaN</Source_3itCtr1_uSER_Test2_crestfactor_Ph3_ratio_FLOAT> 

.USER_Test2_jitterxfer_ratio_FLOAT>1.03</Source_3itCtr1_USER_Test2_jitterxfer_ratio_FLOAT> 

,USER_Test2_BERdegr_dB_FLOAT>0. 756</Source_3itctr1_uSER_Test2_BERdegr_dB_FLOAT> 


Home Insert 
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Clipboard r - 
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fx SourceJitCtrl_StableConfig_TIMESTAMP 



Source _JitCtrl_USER_Test2_BERdegr_dB_FLOAT SourceJitCtrl_StableConfig_TIMESTAMP 

Source_JltCtrl_TimeNow_TIMESTAMP 

Source_JitCtrl_EndedConfig_TIMESTAMP _ 

94 

0.0984 2009/06/19 11:28:00.192 

2009/06/19 11:28:59.808 

2009/06/19 11:26:00.096 

95 

0.232 2009/06/19 11:30:00.288 
2.11 2009/06/19 11:33:00.000 
0 

2009/06/19 11:32:00.384 
2009/06/19 11:35:59.712 

2009/06/19 11:28:59.808 
2009/06/19 11:32:00.384 

30 

97 

98 

99 

0.017189419 2009/06/19 13:48:44.928 
0.046088953 2009/06/19 13:51:35.136 

2UO9/0b/l9 13:50:21.696 
2009/06/19 13:52:56.352 

2009/06/19 11:42:07.776 
2009/06/19 13:50:24.288 

100 

101 

0.093865167 2009/06/19 13:54:14.112 
0.226363662 2009/06/19 13:56:54.816 

2009/06/19 13:55:37.056 
2009/06/19 13:58:17.760 

2009/06/19 13:52:57.216 
2009/06/19 13:55:38.784 

102 

1.176558 2009/06/19 13:59:33.792 

2009/06/19 14:02:38.688 

2009/06/19 13:58:22.080 

103 

0 



104 

0.017189419 2009/06/19 13:48:44.928 

2009/06/19 13:50:21.696 

2009/06/19 11:42:07.776 

105 

0.046088953 2009/06/19 13:51:35.136 

2009/06/19 13:52:56.352 

2009/06/19 13:50:24.288 

106 

107 

0.093865167 2009/06/19 13:54:14.112 
0.226363662 2009/06/19 13:56:54.816 

2009/06/19 13:55:37.056 
2009/06/19 13:58:17.760 

2009/06/19 13:52:57.216 
2009/06/19 13:55:38.784 

108 

1.176558 2009/06/19 13:59:33.792 

2009/06/19 14:02:38.688 

2009/06/19 13:58:22.0S0 

109 

0 

-0.001565549 2009/06/19 14:08:27.744 
-0.00273923 2009/06/19 14:11:44.736 

2009/06/19 14:09:53.280 
2009/06/19 14:13:05.952 

2009/06/19 14:02:43.872 
2009/06/19 14:09:55.008 

1IU 

111 

112 

113 

0.001174218 2009/06/19 14:14:29.760 
0 2009/06/19 14:17:43.296 

2009/06/19 14:15:56.160 
2009/06/19 14:19:24.384 

2009/06/19 14:13:09.408 
2009/06/19 14:16:05.664 

114 

0.008597414 2009/06/19 14:20:51.648 

2009/06/19 14:22:19.776 

2009/06/19 14:19:26.112 

115 

0.019527908 2009/06/19 14:23:40.992 

2009/06/19 14:25:05.664 

2009/06/19 14:22:22.368 

116 

117 

0.05383673 2009/06/19 14:26:26.016 
0.119021154 2009/06/19 14:29:04.128 

2009/06/19 14:27:48.960 
2009/06/19 14:30:38.304 

2009/06/19 14:25:08.256 
2009/06/19 14:27:50.688 

118 

0.439803262 2009/06/19 14:32:02.976 

2009/06/19 14:33:18.144 

2009/06/19 14:30:39.168 

119 

2.229477 2009/06/19 14:34:47.136 

2009/06/19 14:38:51.648 

2009/06/19 14:33:19.872 

!<<►►! Sheetl °J 





Excel 2007 opens 
the XML file. 
DateTimes display 
correctly and can 
be also used 
directly in 
calculations. 

Columns are 
reordered, not 
clear what the 
order is. 

Table sort 
operations actually 
execute as column 
sort. 

Graphing data in 
the table is an 
indirect, text-based 
operation. 



http://grouper.ieee.org/groups/scc20/tii/atml_status.htm 


Prototyping/ATM L 


ATML Overview and Architecture IEEE 
Std 1671-2006 


Signal Descriptions (1641-2004) 



Diagnostics 

AI-ESTATE 

(1232) 


SIMICA 

Maintenance 

(1636.2) 


Test Station (1671.6) 

Instrument 
Description (1671.2) 


Test Adapter (1671.2) 

Test Configuration (1671.4) 

Test Description (1671.1) 


SIMICA Test Results and Session (1636.1) 



ATML Common, Hardware Common, Test Equipment, and Capabilities (1671) 
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Common Data Types 


Prototyping/ATML 



Htx 

Octal 

String 


Binary 

Boolean 

Complex 

dateTime 

Double 


cidoubleArray 


Example_Array |^ | 1 »•» J3— r.1y_Double_Array~^ 


Comment describing yoj- 

root element 


— [+] attributes 


Shall be used by 
element that req. 
specifcabon c* ar 
group's child elem 


esthe 
■ of me 


DefaultElementValue 


a 


<?xml version="1.0" encoding="UTF-8"?> 

<Example_Array 

xsi:noNamespaceSchemaLocation="double_array.xsd" 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
xmlns:c="urn:IEEE-1671:2009.02:Common"> 
<My_Double_Array dimensions="[2,2]" standardllnit- 'V"> 
<c:DefaultElementValue value="0.0" /> 

<c:Element position="[0,0]" value="1.0" /> 
<c:Element position="[l / l]" value="1.0" /> 
<c:Element position="[2 / 2]" value="1.0" /> 
</My_Double_Array> 

</Example_Array> 


& 


The default value of array 
elements. Applies to all array 
elements that are not expbdtfy 
defined through Element child 


! c:Element Ej| — 

•-"-■-“-“-"-"-“-■-“aT- - - 1 1 
0 ..® 

The value of an array 
element. 


^c:double (extension) 


L. 


— [+] attributes 


c:DatumQuality 


luality) ±] 


Shal be used by any 
element that retires the 
specification cf any- of the 
group's chSd elements. 


0 attributes 


position 

The eement's index 


rated by XMLSpy 


www.altova.com 
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TestResults Data Types Prototyping/ATM L 
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The AHA Protocol 

RESTful architecture is simple 

• The state of the LTE is made available to LTE clients through the LTE resource tree 

• It is manipulated through requests to the resource tree 

• The protocol defines the minimum resource tree and standardizes optional branches 
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A Revolutionary New Idea! 
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SCPI 
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SATOCM 


AHA 


Verb Based 
Noun Based 


Probing the AHA Protocol 
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From SQL Server 
Create a connection to a SQL Sewer table. Import data 
into Excel as a Table or PivotTable report. 

From Analysis Services 

Create a connection to a SQL Server Analysis Services cube. 
Import data into Excel as a Table or PivotTable report. 



From XML Data Import 


; Open or map a XML file into Excel. 

■m 


From Data Connection Wizard 

Import data for an unlisted format by using the Data 

Connection Wizard and OLEDB. 

From Microsoft Query 

Import data for an unlisted format by using the Microsoft 
Query Wizard and ODBC. 


Ready 



jsHl EDL 


100 % 


29 2009-ll-18T18:25:17.434999Z Sine 

30 2009-ll-18T18:25:17.559999Z myJD ID_1 

31 2009-ll-18T18:25:17.559999Z backupJD ID_1 

32 2009-ll-18T18:25:17.559999Z Index 
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Ready *3 




Signal Source Generated Sine Wave cl:double 
Index count for call to interface DLL cliinteger 
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want the data headings to appea r , 
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[click an XML mapped cell, ' 
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Human- and Machine-Readable Interface 


Normal Mode Calibration Mode Instrument Setup Device Query 

Normal Mode Calibration Mode Instrument Setup Device Query Setup 



Ubv?e*_ccnUol_cofifig_(ile.xiTil (D:\workXLabRATS_linuAxrri_tentplate) - GVIIV 3 




Edit Tools Syntax Buffers Window Hdp 


(I 1 M::*a fiiefex 
I ftU to* V**-* Hritor, iookmart. looli H«lp 

- C ^ lOtpx/ Vloc albntt A360 fll e/t ertset/c out in l 1 labvm 

- Mod Vnited Getting Started S latest He* ill mas 

Norton Q- 

httpt//l<xainc«t>...et/conuot/latiview 


, Moj.lla f 


r.U £(M Vi*w I h|lory BookmarW Tool* ll*lp 

- O htl|r.//loc «ltMH 1 *OflO'lt»/t»stsrt/statu»yH»ami 

Most Visited uetting Suited v Latest Headlines 




Mlpy/locaBiinl;aO...Uet/»Utin/tiralth 


labview Ite interface 


"Generic Lab VIEW LTE" 
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labview Ite interface 


“Generic LabVIEW LTE" 




Human- and Machine-Readable Interface 



Uestl-1\estl-share\AHA_lnterface.html - Windows Internet ... (^J 


• Documentation 

• Theory of Operation 
•GUI 

• Source-code 

Orchestration resource tre< 


Done 


Web Based! You 

already have the 
tools in your hands! 


. - 1 10 \\Estl-l\estl-share\AH v X 

A & £ \\estl- 1 \estl-share\AHA_I . . . 






Value Proposition: Use Instantly for Box- 
Reuse Later for Subsystem and 


level Development Test 
System Integration Test 
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From SQL Server 

Create a connection to a SQL Server table. Import data 
into Excel as a Table or PivotTable report. 

From Analysis Services 

Create a connection to a SQL Server Analysis Sen/ices cube. 
Import data into Excel as a Table or PivotTable report. 

From XML Data Import 

Open or map a XML file into Excel. 


From Data Connection Wizard 

Import data for an unlisted format by using the Data 

Connection Wizard and OLEDB. 

From Microsoft Query 

Import data for an unlisted format by using the Microsoft 
Query Wizard and ODBC. 


29 2009-ll-18T18:25:17.434999Z Sine 

30 2009-ll-18T18:25:17.559999Z myJD ID_1 

31 2009-ll-18T18:25:17.559999Z backupJD ID_1 

32 2009-ll-18T18:25:17.559999Z Index 
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Signal Source Generated Sine Wave cl:double 


Index count for call to interface DLL cliinteger 
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Other Attributes We Like 



Merges with Industry Trends 
Modular Open Source 
Small but Powerful Command Set 


Rich, Widely Accepted Error-Message Set 
Loosely Coupled 

Metadata, with Data Compression by HTTP 
Security by HTTPS 






Q httpr//localhost: 8080 /. 


4- O ■& httD/'localhostrBOBO/lte/testset/reso’jrces/status 

■*i Graff aanino GTD Slashdot if Google Voice •ft Trick EB £ pacman [Pj Pandora -c* Weather if Td S StomiPose 


O Other Bookmarks 


J □ http://iocalhost:8080/... x * New lab ■- 

<• C rt ft http://localhost:8080/lte/testset/resources/settings 

Graffagnino GTD /, Slashdot if Google Voice ■/! TnckBB Si pacman l‘ Pandora Weather if Tel 


D Other Bookmarks 


7000 


command 


trick Ite interface 


Trick Control 


TRICK LTE Settings 


EDGE version FDGE_v2 l_flFTA_100120_dev 


Host 


EDGE RCS ver 1002L8 


standaloi 


Port 


DOUG display ENGGBAPHICS 


DOUG load 


default 



TestExec LTE and Stackable Orchestration 



Test Master executes Test Scripts 



Test Exec provides Guided Test Flow 


•Scripts are not defined by ICD, scripts developed for 
one Master wouldn't work on a different Master 
•Scripts are not intelligent or visual 
•Scripts are developed by a different process in a 
different language than other software 


Developers: Division of Disciplines 


Process and Data 
lOrchestration 


•AHA Protocol expertise 
•Database expertise 
•IT expertise 


1 LTE and Test Execs | 
•Sensor, Actuator, and 
Simulation tool specialty 
expertise 

•Test and measurement 
methodology expertise 


•Test Exec is a plug-in tool to conduct a specific 
measurement sweep. It incorporates test specialty 
knowledge generally from the same test specialty 
team that develops the LTEs. 

•Test Exec is developed as a specialized LTE. By this 
concept, the same design team can develop LTEs and 
Test Execs by the same processes. 

•Script could invoke a series of Test Execs. 

•Test Orchestrator is in charge, Test Exec provides 
steering by evaluating data, fitting models, 
extrapolating the next point to run ("intelligent"), 
and displaying results and expectations 
("dashboard"). 


Multi-Disciplinary Problem 


NASA has all of the 
necessary disciplines, 
inside the agency! 

• LabVIEW, Trick and 
other Developers and 
Architects at JSC in EV, 
ER, and ESCG 

• Test Orchestration at 
MSFCand JSC 

• METECS has assembled 
a team of part-time 
experts to draw on 

• Test operations users in 
ESTL, AIE; data users in 
Safety 

• Data formats and tools 
developers under CIO's 
office at Ames, JPL, and 
elsewhere 




Intellectua Property protection 


• mouseclick Events and Dialog boxes 
IT Security 

1 ^ kJl ) 

'Vvi»i. ' i «. * f 

• Customer needs today's work done, now 


• Traffic and storage unknowns 

• IT Plans 

• Software Quality 

• Many possible approaches 


Maturity of AHA as a Test and Measurement Protocol 



System Test, Launch 
& Operations 


System/Subsystem 

Development 


Technology 

Demonstration 


Technology 

Development 


Research to Prove 
Feasibility 


Basic Technology 
Research 


10/2010 Objective 


11/2009 

"spiral 2" 

5/2009 

"spiral 1" 


AHA 1/2009 


• AHA is a collection of TRL-9 
protocols. But the resource 
tree, orchestration flow, and 
raw performance need to be 
evaluated in the actual T&M 
end application(s). 


NASA Technology Readiness Levels, applied to Test Orchestration Protocol 


Technology Readiness Level Description 


9. Actual system 'flight proven 1 
through successful mission 
operations 


Protocol has broad acceptance 


8. Actual system completed and 
'flight qualified' through test 
and demonstration (ground or 
space) 


Solid protocol with no TBDs, and good tools exist for easy 
implementation. Penetration into many communities. 


7. System prototype 
demonstration in a space 
environment 


Stable prototype with large-scale integration of the 
prototype into real LTEs in a hardware test environment 
to conduct actual test operations and produce usable 
data products. The protocol is used in several test 
environments and is gaining community acceptance 


6. System/subsystem model or 
prototype demonstration in a 
relevant environment (ground 
or space) 

Stable prototype with small-scale integration of the 
prototype into real LTEs in a hardware test environment 
to conduct actual test operations and produce a real and 
usable data product 

5. Component and/or 
breadboard validation in 
relevant environment 

More stable prototype with small-scale integration of the 
prototype into real LTEs in a hardware test environment 

4. Component and/or 
breadboard validation in 
laboratory environment 

Prototype integration of available open-source 
implementations for selected candidate technologies, in 
a simulation environment, to conduct a simulated test 

3. Analytical and experimental 
critical function and/or 
characteristic proof of concept 

Proof-of-concept integration of available open-source 
implementations for various candidate technologies, in a 
simulation environment 

2. Technology concept and/or 
application formulated 

Technology down-selection, Technical Merit Evaluation 
and Community Acceptance Trending 

1. Basic principles observed and 
reported 

Technology survey, Communities of Practice survey 



Background 
Business Case 
Technical Argument 

BACKUP 



Activities 


Approach 


• Surveyed test technologies used in testing and networking communities 
(DSIL, ISI L, DoD, Commercial) 

• Evaluated these technologies based on our Guiding Principles and ability 
to support the Test Automation Hooks requirements 

• Evaluated several options for discovery and database middleware with 
standalone prototypes 

• Created a minimal set of commands/resources sufficient to implement the 
basic Test Automation Hooks requirements 

• Chose a set of technologies to provide a baseline upon which to further 
evaluate issues and options 

• Prototyped an integrated discovery, commanding, and data gathering 
configuration using this baseline set of technologies and Trick simulations 
to represent test rigs 
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Approach 

Minimal Command/Resource Set 

• Discovery 

- Detect LTE's on network and provide enough info to test manager to identify 
LTE's for test configuration and to define LTE resources 

• Synchronize 

- Add LTE's to test (associate with test_name) and sync time 

• Config 

- reload or change LTE configurations (fully or partially) 

• Data_Transfer 

- Request status data, database logging, file transfer, event messages (associate 
with datajogjd and test_run_id) 

• Exec_Test_Run 

- Start activity associated with test_run_id 

• Stop 

- Stop data logging, test run or test config 

• Custom_Commands/Resources 

- Defined by LTE during identification 
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Test Technology Survey 


Test Technology Survey 
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Test Technology Survey 

Test Technology Survey 


• Performed a survey (web, interviews, tour) to identify 
methods/protocols being used for discovery and test control 
in relevant test communities. 

• Looked at two NASA examples 

- DSIL 

- ISIL 

• Looked at two DoD examples 

- TENA 

- DoD ATS 

• Looked at Commercial Products 

- Nl Labview and Test Stand Test Software 

- IXIA IP Test Automation 
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DSIL 


Test Technology Survey 


• DSIL is a distributed SIL architecture allowing 
coordination and orchestration of remotely located 
SIL assets including coupled dynamics simulations. 

- Flight communication interfaces are through high speed 
data interfaces 

- Simulation communication and synchronization is through 
High Level Architecture (HLA) 

- Orchestration and less time critical Cx Data communication 
between simulations is accomplished with the MAESTRO 
package developed at Marshall 
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DSIL 


Test Technology Survey 


HLA 

- For the DSIL, HLA provides time advancement capabilities used to 
interface coupled dynamics simulation. This capability is not required 
for the Automation Hooks architecture 

- HLA architecture requires a Run Time Infrastructure (RTI) and 
simulation federate interfaces. 

- Although the HLA architecture is well defined, RTFs have their own 
implementations and are in general not interchangeable. Therefore, 
the protocol interface to the Vi's would not be an open standard. 

- Because the Automation Hooks architecture doesn't benefit from the 
advanced features of HLA, the complexity of the architecture does not 
warrant its consideration as a test architecture 
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DSIL 


Test Technology Survey 


• MAESTRO 

- MAESTRO provides orchestration of test commands and has been planned for 
additional use outside of DSIL 

- It is configured using XML and Python scripts 

- Our understanding is that the command tool runs under Windows. Linux 
communication tools exist and have been integrated with Trick simulations 

- Commands are sent using C3I protocol which is a Constellation program 
specific protocol developed for communication between flight and ground 
systems. 

- C3I is not widely supported in the test industry or commercial communities 
and so does not meet our Guiding Principles test for the Automation Hooks 
study 

- It may be of benefit to the Automation Hooks project to look for areas of 
collaboration with the MAESTRO community in the areas of command sets 
and test automation flow. Perhaps MAESTRO can be extended to send 
commands in formats that meet open architecture goals. 
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Test Technology Survey 

ISIL 


• ISIL is a test facility for integration and test of 
ISS software and avionics 

• It is located in Sonny Carter Training Facility 
and its capabilities have grown in concert with 
ISS assembly 

• The facility is now mature. A considerable 
amount of effort has been devoted to 
developing efficient automation interface to 
meet their needs 


May 8, 2009 


Test Automation Hooks Study 
EV- METECS 


38 



Test Technology Survey 

ISIL 


• Communication between test stations (MATE's/Simulators) is 
accomplished via shared memory - not Ethernet 

• Discovery is not required because the bulk of the configuration is well 
known and remains part of the lab assets 

• Personnel involved have a good handle on operation issues in an complex 
and operational facility and would make a good resource for reviewing and 
commenting on potential Automation Hooks pitfalls 

• The software suite used by ISIL is large and complex and it would take 
some effort to determine whether components of it would be of benefit 
to the Automation Hooks project. 

• It may be of benefit to the Automation Hooks project to look for areas of 
collaboration with the ISIL particularly in understanding what test 
automation issues should be considered in planning for an operational 
facility 
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TENA 


Test Technology Survey 


• Test and Training Enabling Architecture (TENA) is a DoD 
initiative supporting test ranges and facilities 

• It is aimed at distributed simulation and test applications and 
hence includes complexity that doesn't contribute to the Test 
Automation Hooks objectives 

• TENA is based on CORBA, an object-based messaging protocol 
that has been declining in popularity because of its complexity 
and historical difficulty penetrating firewalls (interest has 
shifted from CORBA and its competitor DCOM to Web 
Services which are discussed later) 

• TENA relies on middleware that appears to be single source 
and hence does not meet the Guiding Principles 

• For these reasons TENA is not considered appropriate for the 
Test Automation Hooks task 
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DoD ATS 


Test Technology Survey 


• The Navy Automatic Test Systems (ATS) Executive manages the 
Open Systems approach to reduce ATS costs across the DoD 

• ATS has defined an ATS framework that is largely based on the 
following technologies 

- Virtual Instrument Software Architecture (VISA) 

- Interchangeable Virtual Instruments (IVI) 

- Automatic Test Markup Language (ATML) 

• VISA and IVI provide lower level instrument drivers and are 
restricted to Windows OS. They are also not widely used 
outside of the ATE industry. 

• ATML is an emerging XML-based IEEE standard that has 
potential benefit to the Test Automation Hooks task 
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Test Technology Survey 


Commercial Tools 


• Most commercial Automatic Test Equipment provides drivers for test 
instrumentation and interfaces to execute user-provided test scripts. 
Examples include National Instruments Test Stand and IXIA Test Conductor 
(for IP testing) 

• Drivers require Windows so Standard Commands for Programmable 
Instruments (SCPI) commands are used on Linux machines. SCPI is not 
suitable for system level testing 

• The LXI protocol has some interesting discovery capabilities that have 
been evaluated for use in Test Automation Hooks 

• The most interesting development in commercial tools is the emergence 
of XML-based ATML capabilities although these are in the initial stages of 
adoption 
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Technology Assessments - ATML 


Technology Assessments 


ATML 

Discovery 

Messaging 

Database Middleware 
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Technology Assessments - ATML 


Technology Assessment 

ATML 
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ATML 


Technology Assessments - ATML 


• Automatic Test Markup Language promotes test equipment 
interoperability by standardizing around the extensible 
Markup Language (XML) format. 

• Consists of a family of standards and XML schemas that 
represent automate test related information. 

• This includes descriptions of 

- Test configuration 

- Test Stations 

- Instrument Description 

- Unit Under Test Identification 

- Test Adapter 

- Test Description 

- Test Results 

- Diagnostics 
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ATML 


Technology Assessments - ATML 


• The last of the ATML standards is set to be 
released as a trial-use standard 

• One of the standards that is mature and will 
likely survive mostly intact is the Test Results 
(IEEE 1636.1) 

• The ATML standard recommends 
implementation of ATML with use of web 
services and SOAP 
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ATML 


Technology Assessments - ATML 


• A Phase I demonstration of ATML capabilities was 
held at autotestcon in September, 2008 

• A Phase II demonstration is planned for September, 
2009 with the following companies expressing 
interest in participating: Agilent, Boeing, EADS T&S, 
Geotest, Indra, Lockheed Martin, macPanel, National 
Instruments, Northrup Grumman, Pideso, Teradyne, 
TYX, Raytheon, Rohde&Schwarz, Virginia Panel 
Corporation, Vektrex. 
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ATML 


Technology Assessments - ATML 


• The conclusion is that Test Automation Hooks 
architecture would benefit from protocols that 
do not preclude incorporation of ATML as it 
matures 

• This supports adoption of an XML-based and 
web services-based architecture 
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Technology Assessments - Discovery 


Technology Assessment 
Discovery 
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Technology Assessments - Discovery 

Discovery 

• Interested in technologies for discovery on 
Ethernet that allow discovery not only for 
devices, but also of services provided by those 
devices 

• Technologies of interest were VXI-11, UPnP, 
and zeroconf (mDNS/DNS-SD) 
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Technology Assessments - Discovery 

VXI-ll 

• VXI-ll is a common protocol in the T&M 
industry that has discovery capabilities 

• Can be used for the discovery of Ethernet 
connected instruments but has little use and 
support outside of the T&M industry 

• The LXI specification originally mandated VXI- 
ll for discovery but eventually moved to the 
zeroconf discovery method in the 1.2 version 
of the specification 
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Technology Assessments - Discovery 


UPnP 


• A zero configuration networking solution backed by Microsoft and Intel, 
UPnP tries to solve similar discovery issues for small networks without 
centralized servers to manage all devices. 

• Based upon a protocol known as Simple Service Discovery Protocol. Also 
addresses other layers such as addressing, messaging, eventing, and 
presentation. 

• UPnP is becoming common for certain classes of network devices such as 
routers and media servers 

• Has some inertia, is cross platform (uses HTTP as a transport), has a few 
open source implementations/SDKs, and is language neutral. 

• Was considered as the discovery mechanism by the LXI Consortium, but 
the consortium eventually went with zeroconf 

• Does not appear to have any crossover in the T&M industry. 
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Technology Assessments - Discovery 

Zeroconf 


• LXI attempts to reuse a technology developed by Apple called zeroconf (also 
known as mDNS/DNS-SD) to solve the issue of addressing and discovery. 

• mDNS/DNS-SD is a zero configuration networking technology originally developed 
by Apple during their conversion from AppleTalk to IP based networking. It is a 
combination of multicast DNS and DNS Service Discovery. Apple's current 
implementation is known as Bonjour (formerly Rendezvous). 

• Open source implementations exist (such as Avahi for linux) 

• Is cross platform (uses DNS), is language neutral, and has considerable inertia 
(used in many software products, and networked devices such as printers) 

• Is an open standard (specifications available at http://www.multicastdns.org/ and 
http://www.dns-sd.org 

• Has some acceptance by the test and measurement industry: LXI has chosen 
zeroconf as their methods for auto addressing and discovery. Also, many large 
commercial tools (i.e. Labiew, MATLAB, etc.) support discovery of LXI equipment 
using these protocols. 
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Technology Assessment 
Messaging 
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Technology Assessments - Messaging 

Messaging 

• Also referred to as "control" when referring to device 
protocols 

• Messaging technologies are exploding with respect to web 
programming and fields such as Messaging Oriented 
Middleware and Web Services. 

• Many of these technologies were examined (low level, 
protocols, and API level), including XML-RPC, SOAP, CORBA, 
DCOM, JMS, UPnP, Web Services, REST Web Services, AMQP, 
RestMS, etc. 

• Tons of growth and development occurring in this field. May 
be too risky to move to an infant technology, but will want to 
choose a baseline technology that allows growth and 
evolution with these emerging standards. 
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Technology Assessments - Messaging 

UPnP 

• UPnP has its own sense of Control and Event 
messaging. 

• Implemented via SOAP and GENA 

• Meets many of the goals, but does not have 
wide acceptance and does not appear to have 
any use in the T&M industry 
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Technology Assessments - Messaging 

JMS 

• Java Message Service is a Java Message 
Oriented Middleware API 

• Many implementations and tools available 

• Definition at the API level creates 
interoperability issues 

• Not language neutral (although it looks like 
some have attempted wrappers for other 
languages) 
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Technology Assessments - Messaging 

AMQP 


• Advanced Message Queuing Message Protocol 
originated in the financial industry, 

• AMQP defines a wire level protocol ensuring 
interoperability and language independence. 

• AMQP is an open standard, but is in its infancy (1.0 
of the specification is currently in draft status). 

• Message Oriented Middleware solutions include 
capabilities we may never need (message queuing 
and message routing are fundamental to the 
technologies) 
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Technology Assessments - Messaging 

SOAP 

• Originally stood for Simple Object Access 
Protocol 

• Successor of XML-RPC 

• Usually uses XML as its message format and 
HTTP as its transport making it language 
neutral, OS independent, and generally 
immune to firewall issues. 

• Forms the base protocol stack for Web 
Services 
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Technology Assessments - Messaging 

Web Services 


• Defined by the W3C as "a software system designed to support 
interoperable machine-to-machine interaction over a network" 

• Typically uses SOAP and XML with WSDL 

• Web Services Description Language allows a web service provider to 
advertise its commands available and types defined for parameters and 
returns 

• Web Services have gained much inertia from major software vendors, and 
while expected to evolve, should continue to be supported and developed 
around. (WSDL 2.0 was made a W3C Recommendation in 2007) 

• Many implementations and tools available for development 

• OS, Language, and platform independent. 

• The architecture examples in Annex C of the ATML Trial Use Standard 
shows the usage of Web Services and WSDL 
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Technology Assessments - Messaging 

Emerging Technologies 


• With so much growth and evolution occurring in this field, it is important 
to keep an eye on emerging technologies most applicable to test 
automation 

• Web Services over http would allow growth into the following without a 
large shift in architecture (think version numbers): 

- DPWS - Devices Profile for Web Services 

• An implementation and set of rules to allow web service discovery, identification, 
control, and eventing on devices (similar to UPnP) 

- RESTful Web Services 

• Web services implemented adhering to the principles of Representational State 
Transfer 

• Many big web service providers have started to transition from SOAP based Web 
Services to REST implementations 

• Basically a shift from "command-based" to "resource-based" 

• More similar to how http operates. 
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Technology Assessments - Database 


Technology Assessment 
Database Middleware 
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Technology Assessments - Database 

Functionality 

Database-oriented middleware provides the 
following key functions: 

- The ability to convert the application language to 
be understandable by the database 

- The ability to send a query to and process a 
response from a database over a network 
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Technology Assessments - Database 

Objective 

In a test environment, there are likely to be 
many client applications and one database 

- The main goal is to allow the database to be 
upgraded without impacting each client 
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Technology Assessments - Database 

Types of Middleware 

• There are two main types of database-oriented middleware: 

- Native Middleware - created for a specific database 

• Oracle Call Interface (OCI) for Oracle 

• LibPQfor PostgreSQL 

- Call Level Interfaces (CLIs) - provide a single interface to several 
databases 

• Open Database Connectivity (OCBC) is the "gold-standard" for generic 
database interfacing 

• Java Database Connectivity (JDBC) is primarily used for web-based 
applications 

• To avoid making application source code changes on each 
client when the database is upgraded or replaced, a CLI-type 
middleware must be used 
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Technology Assessments - Database 


ODBC 


• ODBC is a multi-database API standard for programs that use SQL to 
access data 

• An ODBC driver for the specific database must be installed on each client 
(this is similar to how a printer driver must be installed before a client can 
print to a specific printer) 
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Technology Assessments - Database 

Bridge Driver 

• An ODBC-ODBC bridge driver provides the capability 
to install the specific driver on the server and keep 
the client side completely generic 
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Technology Assessments - Database 

Compatibility 

• Most Microsoft products contain ODBC interfaces 
(e.g., MS Excel, MS Access) 

• ODBC toolkits are readily available for LabView 

• XML data can easily be stored in any database as a 
text string, however this does not allow for detailed 
searching of the XML field 

• XML to SQL conversion middleware software can be 
used to translate XML data into SQL commands 

• Most databases are now offering native XML data 
types, thus eliminating the need for additional XML 
middleware 
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Technology Assessments - Database 

Database Performance 

• Since ODBC is a higher-level abstraction, there could possibly 
be a performance hit when compared to using a native 
interface 

• A small test was setup using the PostgreSQL database to 
evaluate the performance of using ODBC verses LibPQ, the 
native PostgreSQL interface 

- A Trick simulation was configured to log data to PostgreSQL running on 
a different machine using both LibPQ and ODBC 

- A table is created in the database and then a new row with nine 
double precision variables is added during each logging cycle 

• Test Results 

- It took 0.0012 seconds to log a cycle of data using LibPQ 

- It took 0.0013 seconds to log a cycle of data using ODBC 
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Baseline Technology Set 


Baseline Technology Set 


May 8, 2009 


Test Automation Hooks Study 
EV- METECS 


70 



Purpose 


Baseline Technology Set 


• Based on the technology assessments, a 
baseline set of technologies was chosen for 
discovery messaging, and database 
middleware 

• Purpose of the baseline set is to use as a 
starting point for end-to-end prototyping and 
as a comparison for evaluating other options 


May 8, 2009 


Test Automation Hooks Study 
EV- METECS 


71 



Rationale 


Baseline Technology Set 


• Discovery 

- Zeroconf: based on test industry support and commonality 
with LXI 

• Messaging 

- SOAP: based on XML compatibility (ATML, NExIOM) and 
huge user and support base 

• Database Middleware 

- ODBC to ODBC bridge: based on support by COTS tools and 
desire to reduce LTE maintenance 
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